We'll be leveraging the following third-party tools for our test example:
To illustrate the technique we will leverage
We will be naively performing an exhaustive combinatorial test due to the small app domain.
We'll create a simple .net webapi project consisting of two routes:
The API will produce errors for some value combinations.
In [1]:
# imports
%matplotlib inline
from itertools import product
import json, matplotlib, pyodbc, pandas, uuid, copy
import matplotlib.pyplot as plt
import numpy as np
# some basic data
a = [[True,False],['Purchase','Refinance','LoanModification'],['AZ'],[1,2,3,4]]
pandas.DataFrame(a)
Out[1]:
To cover all combinations of will require n0 * n1 * ... nn combinations
In [2]:
len(a[0])*len(a[1])*len(a[2])*len(a[3])
Out[2]:
Lets see what that looks like by generating all unique combinations.
In [3]:
combinations = list(product(*a))
pandas.DataFrame(combinations)
Out[3]:
We can then convert the combinations to a JSON object which will be our data contract in this example.
In [4]:
for combination in combinations:
valMap = {'isVal':combination[0],'loanType':combination[1],
'state':combination[2],'policyCount':combination[3]}
print(json.dumps(valMap, ensure_ascii=False))
Next, to be able to use our tests in postman we need to create a postman collection. First, we'll import an existing postman collection to a json object.
In [5]:
with open('CombinatorialTesting.json.postman_collection') as data_file:
postman_coll = json.load(data_file)
postman_coll
Out[5]:
Next we manipulate the template using our unique request combinations and dump it back out to the file system.
In [6]:
with open('CombinatorialTesting.json.postman_collection') as data_file:
postman_coll = json.load(data_file)
requestTemplate = copy.deepcopy(postman_coll['requests'][0])
idx = 0;
del(postman_coll['requests'][0])
for combination in combinations:
valMap = {'isVal':combination[0],'loanType':combination[1],
'state':combination[2],'policyCount':combination[3]}
someobj = copy.deepcopy(requestTemplate)
idx+=1
someobj['name'] = str('Example ')+str(idx)
valJson = json.dumps(valMap, ensure_ascii=False)
someobj['rawModeData'] = str(valJson)
someid = str(uuid.uuid4())
someobj['id'] = someid
someobj['url'] = "{{Url}}/Examples"
someobj['method'] = "POST"
someobj['tests'] = str('tests["Status code is 201"] = responseCode.code === 201;')
postman_coll['order'].append(someid)
postman_coll['requests'].append(someobj)
with open('CombinatorialTesting2.json.postman_collection', 'w') as outfile:
json.dump(postman_coll, outfile)